Data logging is the practice of recording data to form log data. Typically, the log data is sequential and is recorded chronologically. In computerized data logging, a computer program automatically records selected events in order to provide an audit trail.
Log mining is the process of analyzing log data for knowledge discovery purposes or for maintaining redundant logical replicas of a database system.
Traditionally, log data was used for database recovery at a physical (block/byte) level. More recently, a variety of business solutions require that log data be translatable into logical (Insert/Update/Delete) SQL operations. Database vendors have gradually begun to add support to extract these changes at a logical level, but this support is not yet available for a limited set of data types. Furthermore, some operations (e.g., large rows) may not be fully logged or not logged at all for efficiency reasons. For example, not logging operations unnecessary for database recovery improves log writing performance, reduces the overhead on the database and thereby reduces commit time latencies. Examples of non-logged data may include bulk loads and updates on large objects. Log data may also contain changes that are fully logged but encrypted for security reasons.
In each of these cases the information in the transaction logs must be supplemented with data from the actual database tables (i.e., row data). That is, information must be fetched directly from data blocks. This may result in inconsistencies between the transaction log and the database, as the database may reflect changes that have not been entered in the transaction log. Thus, there is a problem in any platform where some changes are retrieved from the log and others are fetched from the database.
Therefore, it would be desirable to provide a log mining solution that accounts for missing, partial and/or encrypted log data. More particularly, it would be desirable to provide a log mining fetching solution that guarantees data consistency on transaction boundaries in systems that retrieve data from both a log and a database.